feat: multi-address DNS resolution for contact points and connections (DRIVER-201) - #890

Groundwork for expanding a hostname to all of its addresses: resolution becomes
the connection layer's job, so everything that produces an EndPoint stops doing
DNS of its own, and an endpoint gains a way to record which address a connection
actually reached.

PinnableEndPoint is the new internal contract: pinTo(SocketAddress) returns a
copy bound to one address, and the pin is excluded from equals(), hashCode(),
asMetricPrefix() and toString(). Endpoints are set and map keys, and node
metrics are named after them, so a pinned copy has to be indistinguishable from
its original everywhere except when the connection layer asks which address
answered. A generic delegating wrapper was rejected: its equals() would be
asymmetric, because DefaultEndPoint#equals tests instanceof and would reject the
wrapper while the wrapper accepted the original, and it would break
SniSslEngineFactory's instanceof SniEndPoint guard. Each implementation
therefore carries a nullable pinnedAddress of its own.

SniEndPoint additionally normalizes a resolved proxy *hostname* back to
unresolved. withCloudProxyAddress(new InetSocketAddress("proxy", 9042)) resolves
eagerly, which froze Cloud on whichever proxy IP the JVM happened to return; an
IP-literal proxy is left alone. Contact points keep the opposite policy on
purpose, since ContactPoints.merge() only ever applied its resolve flag to
config-file entries.

ClientRoutesTopologyMonitor.resolve() likewise returns the client route as an
unresolved address and no longer looks it up, which keeps it a pure in-memory
cache lookup that is safe to call from an event loop, and lets a custom resolver
apply to client routes just as it does to contact points. Its protected
resolveAddress() extension point, which existed only so tests could stub out
InetAddress.getByName, goes with it. This has to move together with
ClientRoutesEndPoint: dropping "throws UnknownHostException" from one and the
matching catch from the other is a single compilable change.

TopologyMonitor gains reresolvesNodeAddresses(), which tells the control
connection's reconnection query plan whether this monitor already keeps
addresses fresh. It defaults to false, correct for DefaultTopologyMonitor, whose
peers hold an already-resolved IP from system.peers. ClientRoutesTopologyMonitor
reports true only when every currently-known node actually has a live route:
where the route set is incomplete, ClientRoutesEndPoint falls back to a static
resolved endpoint, and those nodes still need the contact-point fallback.

The "is this a name" test several of these need is shared as
AddressUtils.carriesName(): a resolved address compares its host string against
the literal its own bytes produce, an unresolved one parses its host string.
Neither isUnresolved() nor the presence of an InetAddress can tell a name from a
literal on its own.

DseGssApiAuthProviderBase.serverName() falls back to getHostString() when
getAddress() returns null, which is now the ordinary case for a Cloud or
client-route endpoint rather than an impossible one.

EndPoint.resolve() keeps its signature and is not deprecated, so third-party
implementations still compile. Its javadoc gains two expectations: return the
address as-is rather than looking names up, since this is now called from an
event loop; and callers are warned that the returned address is no longer always
resolved, so getHostString() is the safe way to read the host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…R-201)

This is the fix for DRIVER-201. When a contact point or a node address is a
hostname that maps to several IPs, the driver used to try only the first one and
raise AllNodesFailedException if it was unreachable, even though the hostname
also resolved to healthy addresses.

Resolution is a connection-layer concern. ChannelFactory.connect() is now the
single place that turns "the address this node is known by" into "the addresses
to actually try": EndPoint.resolve() yields one address and does no lookup, so
it stays safe to call from an event loop; ChannelFactory expands it through the
bootstrap's Netty AddressResolverGroup; the candidates are tried in sequence
until one connects; and the endpoint is pinned to the address that won, so the
channel carries the address it is really on.

Expansion always goes through the configured resolver, mirroring Netty's own
doResolveAndConnect0 short-circuit (no group, !isSupported, isResolved) rather
than pre-filtering on isUnresolved(). Both isSupported() and isResolved() are
overridable, so a redirecting custom resolver keeps its say over addresses that
merely look resolved. The bootstrap is built once per connect() and cloned per
attempt, with the clone's resolver disabled: Bootstrap.clone() carries the
resolver over, so an enabled clone would resolve each candidate a second time --
through resolve(), singular -- and a redirecting resolver would collapse every
candidate onto its first answer, silently killing the fallback.

Details that took a round each to get right:

- The queried hostname is re-attached to resolver-returned addresses, centrally
  rather than per endpoint, so TLS sees the name the user configured instead of
  an IP or a CNAME label. Scoped IPv6 keeps its zone via the numeric
  Inet6Address.getByAddress overload; the NetworkInterface one re-derives the
  scope and throws when the interface has no address of the same local type.
- One EventLoop is chosen per connect() and shared by resolution and every
  clone(eventLoop), instead of letting Bootstrap.connect() advance the chooser a
  second time and land channels on half the loops.
- Candidates are shuffled per connect and truncated to the new
  advanced.connection.max-candidate-addresses option (default 5). The shuffle
  spreads load and varies the starting address between attempts with no
  per-name state to maintain; the cap bounds what one attempt can cost -- each
  address tried is a full connect plus handshake, and with wrong credentials a
  rejected login -- while successive attempts sample fresh random subsets, so
  no address is permanently out of reach.
- Protocol-version rejection is terminal only for a node whose host id is known.
  The addresses of an unidentified endpoint may belong to different nodes, and
  collapsing a contact-point hostname into one Node must not lose the query-plan
  advance that resolve-contact-points=true used to provide. An authentication
  failure is never terminal, even for an identified node: with a multi-record
  name it may be specific to the address (a stale record pointing at a foreign
  cluster fails at AUTH, which runs before the cluster-name check), a single
  address's failure must not write off the endpoint, and the candidate cap is
  what bounds the cost of genuinely wrong credentials.
- Failures from earlier candidates are attached to the final error as suppressed
  exceptions -- with the surfaced one chosen by how callers classify errors
  rather than by position -- and negotiation history is scoped per candidate
  address.
- Every resolver and Netty callback completes the connect future on failure.
  connect() has no timeout at the resolution stage, so an unguarded throw would
  hang the caller for good.

afterBootstrapInitialized() now runs once per logical connection rather than
once per attempt, and sees the bootstrap before the driver's handler is
installed; a handler set by the hook is overwritten, with a one-time warning.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…IVER-201)

Node metrics are named after the endpoint, so DefaultNode.setEndPoint() has to
re-register them whenever those names change -- which is not the same question
as whether this is a different node, and the old !equals() test got it wrong in
both directions.

It was too narrow: an unresolved hostname and the resolved address it maps to
compare *equal* while their metric prefixes differ, which is exactly what
happens when a contact-point node adopts the endpoint built from its
system.local row. And too wide in the other direction is now possible too, since
a pinned copy differs from its original only by an address that both equals()
and the metric identity ignore by contract.

The test is therefore asMetricPrefix() plus toString(), because both are in use:
the default MetricIdGenerator names node metrics after the prefix, the tagging
one tags them with toString(). The pin is excluded from toString() as well, or
DefaultTopologyMonitor#buildNodeEndPoint returning the control channel's pinned
endpoint for the system.local row would silently retag node metrics on every
refresh and orphan the old series.

The node also adopts the newest endpoint instance even when it compares equal,
since a pinned copy carries the address every subsequent connection will use.

Finally, the rebuild order is clear, then swap, then build. Dropwizard and
MicroProfile do not remember the ids they registered under; their clearMetrics()
recomputes each one from the node's endpoint as it stands at that moment. The
previous order -- swap, build, clear -- therefore deleted exactly the series the
new updater had just registered and left the old ones behind with nothing
writing to them. That ordering is upstream's, but it used to be reached only
when the endpoints compared unequal; keying the rebuild on metric identity
brings the ordinary contact-point transition onto the same path.

The pre-existing pin test was vacuous: a mocked context yields
NoopNodeMetricUpdater, for which the rebuild is skipped entirely. Both tests now
stub MetricsFactory, and the ordering test drives a real MetricRegistry through
a hostname-to-IP rename; it was proven to fail under the old order.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…e loop (DRIVER-201)

ChannelFactory tries every address a contact-point hostname resolves to while it
opens a channel, but the node's identity was read afterwards, over the channel
that won -- by then the remaining candidates are gone. ControlConnection advanced
its query plan on that failure, and since a contact-point hostname is now a
single Node, that wrote off the whole hostname on the strength of one of its
addresses. With a single contact point and the default reconnect-on-init=false,
session initialization failed outright, and a rebuilt session failed the same
way every time while a healthy address sat unused.

The identity read now happens while the factory still holds the remaining
candidates, through a caller-supplied hook. DriverChannelOptions gains an
internal ConnectHook (precedent for a behavioral member there: eventCallback)
plus a timeout; after protocol initialization succeeds on a candidate,
ChannelFactory invokes the hook and treats a rejection, a synchronous throw or
a timeout as a per-candidate failure: the channel is closed and the loop
advances to the endpoint's next address, exactly like an init failure.

ControlConnection arms the hook only for a node whose host id is unknown --
contact points, the one case with something to learn. The hook runs
TopologyMonitor.getChannelNodeInfo, so a custom monitor's identity read is
honored; it rejects a node that reports no host id, and channels what it read
straight into a per-attempt holder. Once the connect completes, the captured
NodeInfo is registered without a second read, after asserting it came from the
winning channel; a miss (a ChannelFactory subclass that skips the hook) falls
back to a direct read. The options are built fresh per attempt, because the
holder is stateful and overlapping connect chains are reachable: the initial
connect() runs outside Reconnection, and reconnectNow() checks only
initWasCalled. Pool connections and reconnects to identified nodes carry no
hook and send exactly the bytes they sent before.

REGISTER moves out of the init handshake to keep its ordering property:
identity is validated before the channel registers for events. ChannelFactory
sends it through AdminRequestHandler after the hook accepts, with the same
init-query timeout; a registration failure is a per-candidate failure, as it
was as an init step, and the CLIENT_ROUTES_CHANGE rejection keeps its clear
message. The one visible cost: the window in which a live channel is not yet
registered for events grows by the hook's round trip.

The wire cost is unchanged from reading identity after the connect: one
"SELECT * FROM system.local WHERE key='local'" per contact-point connection,
now inside the attempt instead of after it. Init itself ends at the
cluster-name check (or SET_KEYSPACE), byte-identical to the pre-multi-address
exchange, which is what ProtocolVersionMixedClusterIT pins.

Two consequences are visible. A contact point that exhausts every address on
identity failures now fires controlConnectionFailed and is marked DOWN
pre-init, the treatment connect-phase failures already get; and
AllNodesFailedException reports one entry per contact point, with the
per-address failures attached as suppressed exceptions.

One pre-existing exposure stays pre-existing rather than closing: ChannelFactory
imprints the cluster name, product type and negotiated protocol version on init
success, so a candidate the hook then rejects has already imprinted -- like
every other channel abandoned after init (a node turned IGNORED, a close during
resolve). The values are properties of the cluster that answered on that
address, so this is identical to the behavior before this series.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…n (DRIVER-201)

advanced.control-connection.reconnection.fallback-to-original-contact-points now
defaults to true, and is the driver's DNS re-resolution path.

Nothing else re-resolves. Metadata nodes hold an endpoint built from an
already-resolved system.peers IP, and the control node's own endpoint is pinned
by ChannelFactory to the single address its connection reached, deliberately, so
that a node with a known identity cannot wander to a different host. Once the
records behind a hostname change, appending the original contact points is
therefore the only way back: they are still unresolved hostnames, so
ChannelFactory expands each one to its current IPs at connection time.

The append is gated on the topology monitor not re-resolving addresses itself,
since a proxy-based monitor keeps them fresh and raw contact points could
resurrect nodes it has authoritatively removed. The exception is an empty
regular plan: with no live node to try, reconnection cannot recover on its own.

The plans are concatenated rather than mutated. A RUNNING-state query plan is a
built-in QueryPlan whose add()/addAll() throw UnsupportedOperationException,
poll() being its only mutator, so with the fallback defaulting on every
post-init control reconnect would otherwise have crashed. The append is also
skipped before the LBP reaches RUNNING, where newQueryPlan() has already built
the plan from the contact points and appending would duplicate every entry.

Documented cost: the contact points are appended without being compared against
the live-node plan, because at plan time they are hostnames while the live nodes
are resolved IPs. When DNS has not changed they expand to addresses the plan
just failed on, so an exhausted reconnection round retries roughly twice as many
addresses -- which is why HeartbeatIT has to disable it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Rewrites the address-resolution manual page around the connection layer doing
the expansion, and adds an upgrade-guide section covering what changes for
users:

- there is no public API change, but EndPoint.resolve() may now return an
  unresolved address for Cloud/SNI and client-route nodes, so a caller doing
  ((InetSocketAddress) resolve()).getAddress().getHostAddress() gets a NPE where
  it previously worked; getHostString() is the safe read;
- advanced.resolve-contact-points is deprecated and inert;
- fallback-to-original-contact-points defaults to true, with its cost stated;
- a contact point that none of its addresses can identify is now marked down
  before initialization completes, firing an event it did not fire before;
- AllNodesFailedException reports one entry per contact point, with each
  address's failure attached as a suppressed exception;
- the one-time TaggingMetricIdGenerator node-tag rename for hand-built Cloud
  proxy addresses;
- the afterBootstrapInitialized() contract change;
- two protected methods removed from internal classes that a subclass could
  have overridden.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

MockResolverIT drives the end-to-end fix through a JVM-level InetAddress hook:
a hostname that maps to one dead and one live address must still produce a
working session.

Its multi-address test was one change away from being vacuous. The comment
claimed the dead record was tried first because of resolver insertion order, but
rotate() sorts candidates by toString() and discards that order; the dead
address went first only because the sort is lexicographic. The test now captures
ChannelFactory at DEBUG and requires the "trying next address" event, which was
proven load-bearing: moving the dead IP to one that sorts last makes it fail in
7s instead of passing in 89s.

ClientRoutesIT asserts on host strings rather than resolved IPs, since a client
route now stays unresolved until the connection layer expands it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>